Today, the Internet backbone consists of inter-connected networks of major ISPs, government, academic institutions and large private networks. High bandwidth optical links connect these networks at ‘peering’ points. Monitoring these high-speed links can give us a wealth of data. Monitoring has applications in traffic accounting, capacity provisioning, anomaly detection, and fault diagnosis.
In today's Internet, viruses, worms and DDOS attacks are commonplace. Intrusion Detection Systems and Antivirus software detect and stop the spread near the leaf nodes of the victim. It is very beneficial to stop this unwanted traffic even before it reaches the victim's network. On a high-speed backbone network, we have the opportunity to see traffic from multiple networks. This means we have more data available to us and can detect the anomaly well before a small-network administrator. Traditionally, changes in network patterns are used to detect such anomalies.
An important requirement for a service provider is to offer higher availability and fault-tolerance. It is well known that when things go wrong in a network, the traffic can change drastically. For example, an unusual sequence of packets might indicate a router mis-configuration. If we could detect these changes well in advance, we can take preventive measures to ensure that service is not interrupted.
Many traffic engineering problems need accurate measurements of metrics like network demand and heavy-hitters over multiple links. Intra-domain routing protocols like ISIS and OSPF need traffic matrices to assign link weights for optimal routing. These traffic accounting requirements are different from network resource usage measurements. Here, we are interested more in per-customer accounting instead of the network resource as a whole. In addition, this information can be useful in detecting changes in the network usage of customers and offering a better service.
Measuring the usage of network resources can be very useful for network management. Common metrics to measure over the link would involve byte and packet counts, change in packet rate and changes in byte rate. This information, augmented with the statistics from the routers, can give a detailed view of the network health.
With the heightened security focus of governments, ISPs will soon have to comply with laws imposed by governmental bodies. Some laws will require that networks be “wire-tap” friendly and easy to monitor.
Most tier-2 and tier-3 ISPs share each other's networks to carry traffic. This sharing is bound by Service Level Agreements (SLA) between the networks. There is no easy way to enforce or check the adherence to these SLAs at all times. However, one way to do it would be to monitor the traffic on the links.
Most of the tasks (queries) mentioned above are continuous, i.e. they keep examining streams of data continuously instead doing a one-time analysis. Network operators would be interested in running such queries at different points in the network.